fix(server): register the plugin worker manager per process so run dispatch cannot lose it - #306
Open
stubbi wants to merge 1 commit into
Open
fix(server): register the plugin worker manager per process so run dispatch cannot lose it#306stubbi wants to merge 1 commit into
stubbi wants to merge 1 commit into
Conversation
…spatch cannot lose it The plugin worker manager is process-scoped, which is exactly what the failure message says: "sandbox plugin workers are unavailable in this server process". But it was threaded as an OPTIONAL parameter through six runtime construction sites and dozens of route mounts, so any caller that omitted it built a run engine that could never acquire a sandbox lease, and the omission only surfaced when a customer hit that specific path. That has now bitten three times. #304 fixed three route mounts, but four `heartbeatService(db)` calls pass no options at all (routes/activity.ts, routes/instance-settings.ts, routes/summary-slots.ts, services/companies.ts), and heartbeat forwards `options.pluginWorkerManager` straight into `environmentRuntimeService`. Observed in production on 2026-07-26: three runs for one company failed to acquire a lease, dispatched by `assignment` and `automation`, each failing in 0.37s to 0.42s. Sub-second is the tell for a wiring gap; a genuinely absent worker polls for 5s first. Fix the class rather than the instances: register the manager once at app construction and have the sandbox driver fall back to it. That single fallback covers every dispatch path, because they all funnel through createSandboxEnvironmentDriver. An explicitly passed manager still wins, so tests keep injecting their own, and contexts that legitimately have no worker (migrations, CLI) still get the accurate non-retryable error rather than a crash. Chose this over making the parameter required: that is a far larger change across every construction site, and it is still bypassable with `undefined as any`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug, for the third time
Runs fail with:
Seen on production 2026-07-26: three runs for one company, dispatched by
assignmentandautomation, failing in 0.37s to 0.42s. Sub-second is the diagnostic tell for a wiring gap; a genuinely absent worker pollspluginWorkerReadyTimeoutMs(5s) first.Why patching call sites has not worked
The plugin worker manager is process-scoped, which is what the message itself says. But it is threaded as an optional parameter through six
environmentRuntimeServiceconstruction sites and dozens of route mounts. Any caller that omits it builds a run engine that can never acquire a sandbox lease, and the omission is invisible until a customer hits that exact path.#304 fixed three route mounts. It could not fix these, which pass no options at all:
server/src/routes/activity.ts:25server/src/routes/instance-settings.ts:31server/src/routes/summary-slots.ts:31server/src/services/companies.ts:59heartbeatServiceforwardsoptions.pluginWorkerManagerstraight intoenvironmentRuntimeService(heartbeat.ts:5594), soheartbeatService(db)yields a manager-less runtime. That is the pathassignmentandautomationtake.The fix
Register the manager once at app construction, and have the sandbox driver fall back to it:
Every dispatch path funnels through
createSandboxEnvironmentDriver, so one fallback covers all of them, including any construction site added in future.Properties kept deliberately:
isSandboxProviderWorkerUnavailableFailureMessagematches on message text, so retry classification is unchanged.Alternative considered
Making
pluginWorkerManagerrequired in the types would be compile-time enforced, but it is a far larger change across every construction site, breaks contexts that genuinely have no worker, and is still bypassable withundefined as any.Tests
New
server/src/__tests__/plugin-worker-process-registry.test.ts, 5 cases: empty registry returns undefined, registration round-trips, re-registration replaces, an explicitundefinedis treated as absent rather than throwing, and an explicitly passed manager takes precedence over the process registration.